Testing the incremental effectiveness of pay-for-performance to improve implementation of a motivational interviewing brief intervention for substance use disorders in HIV settings: Results of a parallel cluster-randomized type 3 hybrid trial

Background: Substance use disorders (SUDs) have a serious adverse impact on the physical and mental well-being of people with HIV. Previously, using a 39-site dual-randomized type 2 hybrid trial design, findings from the Substance Abuse Treatment to HIV Care Project supported the Implementation and Sustainment Facilitation (ISF) strategy to improve implementation and effectiveness of a motivational interviewing brief intervention (MIBI) for SUD within HIV service settings across the United States (US). Building on this trial, this parallel cluster-randomized type 3 hybrid trial aimed to test the incremental effectiveness of a pay-for-performance (P4P), a form of the “alter incentive/allowance structures” strategy. Methods: Twenty-six HIV service organizations, their staff participants (N=87), and their client participants (N=341) were cluster-randomized to one of two implementation conditions. The control condition included staff-focused training, feedback, and consultation (TFC) and team-focused implementation and sustainment (ISF). The experimental condition included TFC+ISF as well as P4P (TFC+ISF+P4P). P4P used financial incentives to reward MIBI implementation (US$10 per MIBI delivered) and MIBI implementation at or above a pre-defined level of quality (US$10 per demonstration). In addition to these outcomes, past 4-week changes/reductions in client participant’s days of primary substance use and anxiety symptoms were examined. Results: The addition of P4P had a large and significant effect on the number of MIBIs implemented (d=1.30, p<.05) and reduction in anxiety (d=−1.54), but there was no impact on days of substance use. P4P had large effects on MIBI quality (d=1.24) and MIBI implementation effectiveness (d=1.28), but these were not significant (p<.10). Conclusions: P4P is a form of the “alter incentive/allowance structures” strategy. Its function is to reward the implementation of a clinical innovation. Rewarding implementation is consistent with the theory of implementation effectiveness, which suggests implementation climate (i.e., the extent to which implementation is expected, supported, and rewarded) is a key antecedent of implementation effectiveness (i.e., the consistency and quality of implementation). We found that P4P had a significant, positive impact on MIBI implementation in HIV service settings, but client-level outcomes were mixed. Future research should examine the effectiveness of the P4P strategy to improve implementation and sustainment of other evidence-based innovations. Trial registration: ClinicalTrials.gov: NCT04687917. Registered 12/18/2020.


Background
The United States' Ending the HIV Epidemic initiative, announced in early 2019, put forth an ambitious target of achieving 90% reduction in new HIV infections within 10 years [1].Substance use disorders (SUDs) continue to pose a barrier to this goal [refs].For example, in May of 2019, just a few months after the announcement of this initiative and before the COVID-19 pandemic began, Garner and colleagues (2022) [2] conducted a national survey to estimate the prevalence and negative impacts of ve different SUDs among people with HIV (PWH) in the United States.Estimates varied by region of the United States, but overall, average prevalence rates were 42.3% for cannabis, 41.9% for alcohol, 34.6% for opioid, 32.2% for methamphetamine, and 28.1% for cocaine.In terms of the negative impacts that each SUD has on HIV care (e.g., being linked to HIV care, being retained in HIV care) and other important areas of life (e.g., having stable housing, being employed), results of the national survey found the most problematic was methamphetamine and the least problematic was cannabis.When the prevalence rate and negative impact estimates were multiplied to estimate the population-level negative impact, the top three most problematic use disorders were alcohol, methamphetamine, and opioids.Because COVID-19 has been reported to have increased the use of alcohol and other substances as well as the prevalence of anxiety [3][4][5][6], the rates of comorbid HIV, SUD and mental health problems may now be even greater.
HIV health and social service settings provide a critical yet under-utilized opportunity to address these adverse impacts [7][8][9][10].According to a review by Hitch and colleagues (2019) [11], SUD screening within HIV settings is limited.In addition to highlighting the need for HIV settings to conduct systematic SUD screening using validated tools, they noted screening must be accompanied by strategies to facilitate delivery of brief interventions and referral to SUD treatment [11].This recommendation is consistent with other calls to improve the integration of SUD services within HIV settings [7][8][9][10].More recently, as part of its Overdose Prevention Strategy, the Department of Health and Human Services appropriated $43.7 million dollars of funding in 2022 to support provision of accessible, evidence-based, and culturally appropriate SUD services for people with HIV or at high risk for HIV [12], and further HIV-focused SUD care allocations can be anticipated.The current implementation research, which focused on testing the incremental effectiveness of a pay-for-performance (P4P) strategy to help improve the implementation of an empirically-supported motivational interviewing brief intervention (MIBI) for SUD within HIV settings [13], may help inform how such funding is utilized.The current research may also help inform the eld of implementation science, which continues to need effective strategies for improving the implementation of evidence-based innovations within real-world practice settings.
The Substance Abuse Treatment to HIV Care (SAT2HIV) Project In 2014, as part of its effort to address comorbid HIV and SUD, the National Institute on Drug Abuse funded the Substance Abuse Treatment to HIV Care (SAT2HIV) Project.Using a dual-randomized type 2 hybrid trial design, 39 HIV service organizations (HSOs) were randomized to one of two implementation conditions [14] and within each HSO, client participants were randomized to one of two intervention conditions [15].The control implementation condition, based on research by Miller and colleagues (2004) [16], was a staff-focused strategy that provided 2 staff per HSO with training, performance feedback, and on-going consultation in implementing the MIBI with client participants.This is the strategy used by the Addiction Technology Transfer Center (ATTC), an intermediary organization funded by the Center for Substance Abuse Treatment, to prepare staff for implementing motivational interviewing.In addition to this strategy, a team-focused strategy called Implementation and Sustainment Facilitation (ISF) was provided to HSOs and their participating staff (2 MIBI staff and 1-3 leadership staff) randomized to the experimental condition.Grounded in the theory of implementation effectiveness [17][18][19], which posits implementation climate (i.e., the extent to which implementation is expected, supported, and rewarded) is a key mechanism of change, the ISF strategy focused on optimizing the support dimension of implementation climate during monthly 30-60 minute meetings with the HSO's MIBI staff and Leadership staff.These meetings were facilitated virtually by an ISF facilitator.Consistent with the Exploration, Preparation, Implementation, and Sustainment (EPIS) framework [20,21], the focus of ISF meetings varied by EPIS phase.All ISF meetings, however, were guided by the principles of motivational interviewing: engaging, focusing, evoking, and planning [22][23][24].
The control intervention condition was usual care only (i.e., passive referral [e.g., pamphlet] to a local treatment provider), and the experimental intervention condition consisted of usual care plus a single 15 to 30-minute MIBI.As detailed by Garner and colleagues (2020) (11), compared to the staff-focused training, feedback, and consultation (TFC) control strategy, the addition of the ISF Strategy (TFC + ISF) resulted in signi cantly greater levels of implementation; both a greater number of MIBIs being implemented and a greater level of MIBI quality (i.e., delity) scores.Moreover, compared to usual care only, usual care plus the MIBI signi cantly reduced daily use of their primary substance for clients within the TFC + ISF condition but not within the TFC only condition [13].

Rationale for Testing Pay-For-Performance
Notwithstanding SAT2HIV Project's success supporting ISF, signi cant room for improvement in implementation effectiveness remained.The rationale for testing P4P was twofold.First, P4P directly addresses the reward dimension of implementation climate -a key mechanism of change according to the theory of implementation effectiveness [17][18][19].Second, prior implementation research supported P4P as highly effective and cost-effective for improving implementation of an evidencebased SUD intervention for adolescents [25][26][27].Notably, P4P is a form of Powell and colleagues' (2015) [28] "alter incentive/allowance structures" strategy they de ned as "work to incentivize the adoption and implementation of the clinical innovation."

Rationale for Trial Design
The current study used a parallel cluster-randomized design (staff clustered within HSO [the unit of randomization]) to minimize the likelihood of contamination across the project's two implementation conditions, and because the parallel cluster-randomized design may be preferable to alternatives, including the stepped-wedge trial design [29][30][31][32].In addition to the use of a parallel cluster-randomized design, a type 3 hybrid trial was used because of the importance of examining both implementation outcomes and client outcomes [33][34][35][36][37], including anxiety symptoms that are strongly associated with SUDs among people with HIV [38] and which have potential for improvement following SUD intervention [39,40].

Aims and Hypotheses
As illustrated in Fig. 1, the aims of this parallel 26-site cluster-randomized type 3 hybrid trial included testing the effectiveness of P4P for improving implementation outcomes as well as client outcomes.We hypothesized that relative to MIBI staff trained and supported using the TFC + ISF strategy (control condition), which had been the most effective and cost-effective strategy in the prior trial [13,41], that MIBI staff trained, supported, and rewarded using the TFC + ISF + P4P strategy (experimental condition) would achieve signi cantly greater: (a) MIBI implementation consistency, (b) MIBI implementation quality, (c) MIBI implementation effectiveness (i.e., the consistency and quality of implementation), and client participants in the experimental condition would achieve signi cantly greater (d) reductions in days of primary substance use, and (e) reductions in anxiety symptom severity.

Trial Design
The design was a 26-site parallel cluster-randomized type 3 implementation-effectiveness hybrid trial [34,37].Guided by the EPIS framework [20] HSOs (and their staff) were recruited and randomized to one of two conditions (TFC + ISF or TFC + ISF + P4P) during the exploration phase.During the hybrid preparation-implementation phase the TFC + ISF strategy was available to HSOs (and their staff) in both conditions.The P4P strategy was available to MIBI staff cluster-randomized to the TFC + ISF + P4P condition from onset of the preparation-implementation phase.A MIBI staff transitioned to implementation by implementing a MIBI with a client participant.Thus, HSO's were in a hybrid preparation-implementation phase until all MIBI staff had implemented the intervention with at least one client participant.The preparation-implementation phase was 12-months for the project's rst cohort and 6-months for the project's second cohort.This unplanned difference was due to the project being stopped early when the principal investigator changed institutions.Approval and oversight of all research activities was provided by the Institutional Review Board of Advarra.

Context
The project's participating HSOs (N = 26) were located in 13 states and the District of Columbia within the United States.

Participants
Staff participants.For an HSO to be eligible for the project, it had to receive funding from the Health Resources & Services Administrations' Ryan White program, have two or more staff members (e.g., case manager, behavioral specialist) willing to be trained to implement a MIBI for SUD, and have at least one leadership staff (e.g., director, manager, supervisor) willing to help ensure the MIBI staff were provided su cient time for participation.The participating HSOs identi ed individual staff to be invited to participate.Staff were required to be at least 18 years of age.There were not any exclusion criteria for HSO or staff participation.After staff provided informed consent to participate in the project, which was obtained electronically, they completed an online survey.Each participant received a US$25 e-gift card upon completion.The survey assessed background characteristics (e.g., age, biological sex, ethnicity, race, highest level of education), implementation climate using the six-item measure developed and described by Jacobs et al. (2014) [42], motivational interviewing knowledge and experience using a twoitem index measure developed for this study and where staff self-rated their current baseline knowledge (0 = no knowledge at all, 1 = knowledgeable to a small extent, 2 = knowledgeable to a moderate extent, 3 = knowledgeable to a great extent) and experience (0 = none, 1 = beginner, 2 = intermediate, 3 = advanced, 4 = expert) using MI, setting-intervention t using the six-item measure developed and described by Garner et al. (2022) [43], and organizational readiness for change using the 12-item measure developed and described by Shea et al. (2014) [44].
Client participants.To be eligible, clients had to be at least 18 years of age and HIV positive.There were no exclusion criteria for client participation.Client participants were recommended to receive a MIBI when they reported unhealthy alcohol use or endorsed two or more of the 11 Diagnostic and Statistical Manual of Mental Disorders (DSM-V) criteria [45] for at least one substance during the past 12 months.Client participants received a US$10 e-gift card upon completion of the online SUD screener.MIBI-recommended client participants were asked to complete additional online assessments.These included a pre-MIBI assessment completed within the 24-hour period prior to receiving the MIBI and a follow-up assessment completed 4 weeks after receiving the MIBI.The MIBI was provided at no cost.Participants received e-gift cards upon completion of each assessment (US$20 for pre-MIBI assessment; US$10 for 4-week follow-up assessment).For the 4-week follow-up assessment, participants received a US$10 bonus for completion within 24 hours of the online link being emailed or texted to them or a US$5 bonus for completion within 25-48 hours of the online link being emailed or texted to them.

Strategies
As part of the SAT2HIV Project, the team-focused ISF Strategy was an effective adjunct strategy to the staff-focused TFC Strategy (ISF + TFC) [13].As such, ISF + TFC was the control strategy and ISF + TFC + P4P as the experimental strategy in the SAT2HIV-II Project.Regarding temporality, ISF was the strategy HSOs and their MIBI staff received rst.This was strategic and designed to help optimize MIBI staff's early understanding and motivation for the TFC Strategy's components.Importantly, for HSOs randomized to ISF + TFC + P4P, the ISF strategy was intended in part to accelerate MIBI staff's understanding and motivation for achieving the P4P Strategy targets: MIBI consistency and MIBI Quality.Each strategy, described below and in the Table 1.The Strategies Timeline, Activities, and Resources (STAR) Table is a novel tool that includes a summary of information logged as part of the Strategies Timeline, Activities, and Resources (STAR) Log (see https://www.c-dias.org/implementationguides-and-measures/).Implementation and sustainment facilitation.The ISF is a team-focused strategy that seeks to optimize MIBI consistency and MIBI quality via elevating implementation climate (i.e., the extent to which MIBI staff perceive these dimensions of implementation effectiveness to be expected, supported, and/or rewarded) [13,14].It is team-focused in that it requires engagement of staff to deliver the intervention as well as leaders to help address implementation barriers (e.g., competing priorities, insu cient time).To help optimize staff understanding of the implementation initiative and motivation for the training provided as part of the preparation phase, it is recommended that the rst ISF strategy meeting be completed prior to the initiation of the staff-focused training strategy.Consistent with other research that has used motivational interviewing as an implementation strategy [46,47], ISF uses key principles of motivational interviewing (i.e., engaging, focusing, evoking, planning) as a guide.
In this project, each HSO engaged in a virtual monthly ISF strategy meeting that included three or more of the four motivational interviewing principles.These meetings lasted, on average, 30-60 minutes.An ISF workbook programmed in Microsoft Excel helped standardize the ISF strategy, document HSO staff attendance, document which motivational interviewing principles were employed, and document any meeting notes and/or action items.The ISF strategy was provided to each HSO by one of four master-level facilitators, two of which had provided ISF as part of the SAT2HIV Project.ISF meetings were recorded and a link to the meeting recording was sent to the HSO's MIBI staff and leadership staff following each meeting.Weekly ISF team meetings supervised by a PhD-level ISF strategy facilitator were conducted for quality assurance purposes.
Training, feedback, and consultation.The TFC strategy, similar to the ATTC strategy in the SAT2HIV Project [13,14], is a stafffocused strategy composed of three discrete strategies.As part of the current project, MIBI staff were provided access to the project's MIBI manual, Feedback on the quality of MIBI implementation was provided via a machine-learning based feedback system for motivational interviewing [48].For each MIBI session at least 10 minutes in length, a MIBI quality score was recorded.The range for each MIBI quality score was 0 to 12, with higher scores indicative of higher quality.MIBI staff were required to demonstrate at least average MIBI pro ciency (a score of 4) via a practice role play session, which was based on the average MIBI pro ciency provided as part of the MIBI effectiveness trial [13], hence su cient for achieving intervention effectiveness.MIBI role play sessions were conducted with another MIBI staff at their HSO who played the role of the client using a standardized client scenario provided by the project.
In addition to training and feedback, MIBI staff were provided the opportunity to participate in group MIBI consultation meetings.Conducted separately for each condition, these meetings were provided monthly, were 30-60 minutes in length, and were conducted by the same trainer who conducted the virtual training sessions.In addition to providing didactic booster training regarding motivational interviewing and our project's MIBI protocol, these group consultations provided MIBI staff with the opportunity to ask speci c questions about how to improve the quality of MIBI implementation with client participants.Each consultation meeting was recorded and a link to the recording was emailed to the MIBI staff in the respective condition.
Pay-for-Performance. P4P is a staff-focused strategy that utilizes nancial incentives and explicitly targets the reward dimension of implementation climate -the extent to which implementation is expected, supported, and rewarded [17][18][19].For each MIBI session implemented, the MIBI staff received a US$10 incentive.Additionally, for each MIBI session at least 10 minutes in length and that received a 6 + quality rating via a machine-learning based feedback system for motivational interviewing [48], the MIBI staff received an additional US$10 incentive.This machine-learning based system enabled MIBI staff to receive immediate feedback on session quality, including whether their overall score met or exceeded the project's pre-de ned quality target.The project's target was a score of 6, which was the 80th percentile score obtained by MIBI staff as part of the MIBI effectiveness trial described above [13].
For the project's rst cohort, P4P incentives that incentivized attendance at the group consultation meetings and ISF meetings were introduced after month 6.Approved by the project's IRB, MIBI staff were asked to complete a new informed consent form to document their consent to receive a US$30 incentive per monthly group consultation meeting attended and a US$30 incentive per ISF meeting attended.Also approved by the IRB, leadership staff were asked to complete a new informed consent form that documented their consent to receive a US$30 incentive per ISF meeting attended.These additional P4P incentives were only for the HSOs and their participating staff that were randomized to the TFC + ISF + P4P condition.This revised P4P protocol was offered from the start for the project's second cohort.During the rst week of each month, research staff calculated the incentive amount earned by MIBI staff during the prior calendar month, sent a noti cation email to the staff, and had an e-gift card for the total incentive amount earned sent to the MIBI staff.There was not a limit on the amount MIBI staff were able to earn.
[Table 1] Intervention The project's single session MIBI protocol has been shown to be effective in multiple settings [13,49].It was designed to help motivate an individual with comorbid HIV and SUD to change their primary substance use by: (a) examining their reasons for change, (b) receiving feedback about some common negative interactions between substance use and HIV-related physical and mental health issues, (c) increasing the importance or con dence to reduce or stop their primary substance use, and (d) making a plan for change [15].

Outcomes
Implementation outcomes.According to the theory of implementation effectiveness [17][18][19], implementation effectiveness is a multidimensional construct representing the consistency and quality of implementation by targeted users.Consistency is similar to measures like reach [50] and penetration [51] and quality is similar to measures like delity [51] and integrity [52].MIBI consistency was operationalized as the total number of MIBIs a trained MIBI staff implemented during the project, with no minimum session length.MIBI quality was operationalized as the total quality score a trained MIBI staff demonstrated during the project.For a MIBI quality score to be generated a MIBI session had to be at least 10 minutes in length.Each MIBI at least 10 minutes in length received a quality score between 0 and 12, with higher scores indicative of higher quality.Following the standardization of each respective measure, these two measures were summed and standardized to create a measure of implementation effectiveness (i.e., the consistency and quality of implementation) [17][18][19].
Client outcomes.Consistent with its type 3 hybrid trial design, this project also examined the incremental impact of P4P on client outcomes 4-weeks post-MIBI.The two client outcomes of interest were change in days of primary substance use and change in anxiety symptom severity, both of which were based on client participant's self-report.They were asked "out of the past 28 days

Participant ow and recruitment
Two cohorts of HSOs were recruited and randomized as part of the project.The rst cohort, which cluster-randomized 12 HSOs, 44 MIBI staff, and 198 client participants, occurred from January 2021 through December 2021.The second cohort, which cluster-randomized 14 HSOs, 43 MIBI staff, and 143 client participants, occurred from November 2021 to April 2022.Figure 2 details the ow of the project's 26 HSO (87% of targeted sample), 87 MIBI staff (97% of targeted sample), and 341 client participants.The targeted sample size of 30 HSOs and 90 MIBI staff was not achieved because the project's principal investigator changed organizations and the trial was forced to stop early.A target number of client participants was not prespeci ed because number of MIBIs implemented (MIBI consistency) was a primary outcome of interest.
[FIGURE 2] (4 weeks), about how many days did you use [insert their primary substance]."The Generalized Anxiety Disorder 7-item (GAD-7) [53], with scores ranging from 0 to 21 (lowest to highest severity), was used to assess anxiety during the past two weeks.These client outcomes were aggregated for each MIBI staff.Thus, for each trained MIBI staff we computed two client outcome impact measures.One for their impact on days of primary substance use, and one for their impact on anxiety.A change score was computed for each possible client participant (i.e., those who completed the 4-week follow-up assessment) by subtracting their respective pre-MIBI assessment measure from their respective MIBI follow-up assessment measure.Then, for each respective change score measure, we aggregated to the level of the corresponding MIBI staff.Negative values are preferable as they represent greater reductions in the outcome.If a MIBI staff member was trained but implemented the MIBI with zero client participants, their client impact scores were zero (i.e., no impact, no return on investment).

Targeted sample size
The targeted sample size was 30 HSOs and 90 MIBI staff.Assuming an intraclass correlation of .06,power analysis indicated su cient power to detect medium-sized effects.

Randomization sequence generation
For each project cohort, randomization of HSOs (the clusters) was completed by the project's principal investigator, project coordinator, and statistician.The principal investigator and project coordinator independently matched pairs of HSOs based on information collected as part of an organizational background form (e.g., number of people with HIV served per year, number of staff employed, number of staff who have been trained in any type of motivational interviewing).They then met to compare rankings and reach consensus on a nal list of match pairs.The nal list of matched pairs was given to the statistician who randomized each matched pair to one of the two implementation conditions.

Blinding (Masking)
Client participants were blinded (masked) to implementation condition, as was the MIBI training staff.There was no blinding (masking) of implementation condition for the HSOs and their staff participants, or any other members of the study team.

Statistical methods
Statistical analyses were conducted using an intention-to-implement approach.All MIBI staff who received some component of the TFC strategy or ISF strategy were included in the analyses.All analyses were conducted using SAS version 9.4 [54].An initial analytic step was to examine the partitioning of variance for each outcome measure.Mixed effects regression analyses were used for the impact of P4P on implementation outcomes and general linear regression analyses were used for the impact of P4P on client outcomes.Doubly robust estimation (i.e., combined propensity score weighting with outcome regression) was used to increase the models robustness to misspeci cation [55][56][57][58].Factors used to create the staff propensity weight were age, gender, ethnicity, race, education, and experience, both with the organization and in their current position with the organization.
All analyses controlled for project cohort.We examined the extent to which there were any signi cant interactions between condition assignment and the staff factors included in the model.If no signi cant interaction was found, we focused on the main effect of the implementation condition.
Half of the 26 HSOs were randomized to TFC + ISF and the other half were randomized to TFC + ISF + P4P.In the TFC + ISF condition, one HSO withdrew from the project after randomization and before any of the project's preparation phase strategies (ISF meetings, MIBI training) could be provided.The number of MIBI staff per condition was nearly the same, with 44 in TFC + ISF and 43 in TFC + ISF + P4P.The number of client participants in the TFC + ISF + P4P condition (n = 224) was 91% more than in the TFC + ISF condition (n = 117).The positive screening rate was 14 percentage points higher in TFC + ISF + P4P (80%) than in TFC + ISF (66%).There was little difference between conditions in terms of the number of client participants who withdrew from the study prior to completing the pre-MIBI assessment, with this rate being low for both conditions (10% for TFC + ISF and 8% for TFC + ISF + P4P).
In addition to the SUD screening assessment, client participants were asked to complete a pre-MIBI assessment within the 24hour period prior to receiving the MIBI.If the pre-MIBI assessment and MIBI were not completed within 30 days of the SUD screen, the MIBI participant was documented as lost.The percentage of client participants lost between completion of the SUD screener and the pre-MIBI assessment was 18 percentage points lower in TFC + ISF + P4P (30%; 50 of 165) than in TFC + ISF (48%; 33 of 69).Rates of completion for the 4-week post-MIBI assessment were nearly identical between conditions: 66% (23 of 35) for the TFC + ISF condition and 67% (70 of 104) for the TFC + ISF + P4P condition.There were no signi cant differences between clients retained and clients lost to follow-up.

Participant characteristics
Table 2 presents the baseline characteristics for MIBI staff participants for the overall sample (N = 87) and each condition (43 for TFC + ISF; 44 for TFC + ISF + P4P).Most MIBI staff were female (63%), non-Hispanic (77%), non-White (63%), and did not have a graduate degree (72%).On average, MIBI staff were 40.0 (SD = 12.5) years of age, had 51.4 (SD = 67.6)months of experience with their HSO, and 37.3 months (SD = 53.1) of experience in their current position.Table 3 presents the characteristics of client participants who were recommended for and received the project's MIBI overall (N = 139).See additional les for the characteristics of all client participants who consented to participate in the study and completed the SUD screener (N = 341) and the subset who were recommended for a MIBI (N = 257).[Tables 2 & 3] Strategy dosage  [Table 4]
[Tables 5 & 6] DISCUSSION There is growing interest in improving the integration of SUD services within HIV service settings [11,9,[59][60][61].Consistent with a systematic review highlighting the need for studies that "move beyond discrete training events and towards longer term coaching-type activities focused on implementation" [62] Garner and colleagues (2020) [13] found adding the ISF strategy (a team-focused coaching-type strategy) on top of the TFC strategy (a staff-focused training-type strategy) signi cantly improved both (a) the implementation of a MIBI for SUD within HSOs, and (b) the effectiveness of the MIBI in reducing substance use.
Building on this research, the aim of the current 26-site parallel cluster-randomized type 3 hybrid trial was to test the incremental effectiveness of P4P (TFC + ISF vs. TFC + ISF + P4P) on both implementation outcomes and client outcomes.
In the current study, addition of P4P had a large and statistically signi cant impact on MIBI consistency (the number of MIBIs implemented), with the TFC + ISF + P4P condition implementing more than triple the number implemented within the TFC + ISF condition: 115 vs. 36.This large effect of P4P on implementation is consistent with prior research that found P4P to improve implementation of an adolescent substance use intervention [26].
The addition of the P4P strategy also had a large effect on the overall level of MIBI quality achieved by MIBI staff, but this effect was not statistically signi cant according to our pre-speci c criteria of p < .05.However, we note that several methods-focused publications have recently argued against the use of this cutoff [63-65].Rather, in contrast to the statistically signi cant effect found for MIBI consistency, the impact of P4P on MIBI quality was slightly smaller and short of achieving statistical signi cance due to insu cient statistical power because of having insu cient sample size.This was also the case for MIBI implementation.
Our nding that P4P had a slightly less positive effect on quality than on quantity is in line with ndings from a recent metaanalysis by Kim and colleagues (2022) [66].More speci cally, the meta-analysis by Kim et al. found that "the effect of incentives on performance was consistently and meaningfully larger for performance quantity than for quality," which is a nding they highlighted as being consistent with cognitive evaluation theory [67-69] and its later incorporation into self-determination theory [70][71][72][73].
Regarding impact on client outcomes, MIBI staff in the TFC + ISF + P4P condition had slightly greater overall changes/reductions in days of primary substance use by clients compared to the TFC + ISF condition (d = − .34),but this was not at all close to being statistically signi cant (p = .59).Although also not statistically signi cant (p = .11),we did nd that MIBI staff who had more experience with their organization had greater overall changes/reductions in days of primary substance use by clients.However, given that more experience working in the HSO was associated with poorer MIBI quality (though not statistically signi cant), we believe it may be possible that MIBI staff with greater experience working in their HSO were able to establish stronger therapeutic rapport/alliance with their clients; a relationship that much research has found to be one of the best predictors of improved client outcomes.Although beyond the scope of the current research, it may therefore be warranted for future research to rate MIBI sessions for therapeutic rapport/alliance and to examine its relationship with client changes/reductions in days of substance use.
We did nd that client participants in the TFC + ISF + P4P condition had signi cantly greater changes/reductions in anxiety symptom severity (d = -1.54;p = .04)compared to those in the TFC + ISF condition.This nding is encouraging regarding the potential bene t of SUD intervention on anxiety outcomes.It must also be acknowledged that the more signi cant predictors of this outcome was if the MIBI staff was female (p = .03)or Hispanic (p = .007).There is research that has previously suggested a "female effect" [74] in better mental health outcomes, but we are not aware of research that has found similar effect for interventionists who identify as Hispanic.Future research exploring factors associated with improvement in anxiety may therefore be warranted. limitations The current research was not without limitations.First, both implementation conditions were affected, conducting the research during COVID-19 was not ideal and may have limited the extent to which the MIBI was able to be implemented.Second, the trial's sample size and associated statistical power was lower than targeted, which as noted above in the methods section was due to the trial being stopped earlier than planned.Third, client outcomes were limited to two measures based on self-report.Fourth, Legend not included with this version Legend not included with this version

Supplementary Files
This is a list of supplementary les associated with this preprint.Click to download.

AdditionalFile1CONSORTChecklist.docx
AdditionalFile2StaRIChecklist.docx P4P Tracker Note: ISF = Implementation & Sustainment Facilitation; P4P = Pay-for-performance; HSO = HIV Service Organization; MI = Motivational Interviewing; MIBI = Motivational Interviewing Brief Intervention 4-hour online introduction to motivational interviewing training course, and 12-hour virtual training in the MIBI protocol.Although conducted by the same trainer as the SAT2HIV Project, who was and remains part of the Motivational Interviewing Network of Trainers, the 12-hour virtual training was a condensed version of the 2-day in-person training used in the prior trial.This approach was necessitated due to COVID-19 safety protocols.The training content provided as part of the project's in-person training was condensed into three 4-hour virtual training sessions and was conducted over a four-week period.A one-week break between the second and third training sessions was provided to enable MIBI staff with time to practice the MIBI before the third training session.Free continuing education credits were provided for completion of both the online training and virtual training.

Table 4
Strategy dosage for MIBI staff in the two conditions for each project rollout.

Table 5
Results of analyses of the impact of the P4P strategy on implementation outcomes